Self-calibrated, remote imaging and data processing system

ABSTRACT

An imaging sensor system, adaptably mountable to a vehicle having a view of a target area comprising: a rigid mount unit having at least two imaging sensors disposed within the mount unit, wherein a first imaging and a second imaging sensor each has a focal axis passing through an aperture in the mount unit, wherein the first imaging sensor generates a first image area comprising a first data array of pixels and the second imaging sensor generates a second image area comprising a second data array of pixels, wherein the first and second imaging sensors are offset to have a first image overlap area in the target area, wherein the first sensors image data bisects the second sensors image data in the first image overlap area.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.12/798,899, filed on Apr. 13, 2010, which was a continuation-in-part ofU.S. patent application Ser. No. 11/581,235 (now issued as U.S. Pat. No.7,725,258), filed on Oct. 11, 2006, which was a continuation-in-part ofand claimed priority to U.S. patent application Ser. No. 10/664,737 (nowissued as U.S. Pat. No. 7,127,348), filed on Sep. 18, 2003, whichclaimed priority to U.S. Provisional Patent Application Ser. No.60/412,504, filed on Sep. 20, 2002 for “Vehicle Based Data Collectionand Processing System.

TECHNICAL FIELD OF THE INVENTION

The present invention relates, generally, to the field of remote imagingtechniques and, more particularly, to a system for renderinghigh-resolution, high accuracy, low distortion digital images over verylarge fields of view.

BACKGROUND OF THE INVENTION

Remote sensing and imaging are broad-based technologies having a numberof diverse and extremely important practical applications—such asgeological mapping and analysis, and meteorological forecasting. Aerialand satellite-based photography and imaging are especially useful remoteimaging techniques that have, over recent years, become heavily relianton the collection and processing of data for digital images, includingspectral, spatial, elevation, and vehicle location and orientationparameters. Spatial data—characterizing real estate improvements andlocations, roads and highways, environmental hazards and conditions,utilities infrastructures (e.g., phone lines, pipelines), andgeophysical features—can now be collected, processed, and communicatedin a digital format to conveniently provide highly accurate mapping andsurveillance data for various applications (e.g., dynamic GPS mapping).Elevation data may be used to improve the overall system's spatial andpositional accuracy and may be acquired from either existing DigitalElevation Model (DEM) data sets or collected with the spectral sensordata from an active, radiation measuring Doppler based devices, orpassive, stereographic calculations.

Major challenges facing remote sensing and imaging applications arespatial resolution and spectral fidelity. Photographic issues, such asspherical aberrations, astigmatism, field curvature, distortion, andchromatic aberrations are well-known problems that must be dealt with inany sensor/imaging application. Certain applications require very highimage resolution—often with tolerances of inches. Depending upon theparticular system used (e.g., aircraft, satellite, or space vehicle), anactual digital imaging device may be located anywhere from several feetto miles from its target, resulting in a very large scale factor.Providing images with very large scale factors, that also haveresolution tolerances of inches, poses a challenge to even the mostrobust imaging system. Thus, conventional systems usually must make sometrade-off between resolution quality and the size of a target area thatcan be imaged. If the system is designed to provide high-resolutiondigital images, then the field of view (FOV) of the imaging device istypically small. If the system provides a larger FOV, then usually theresolution of the spectral and spatial data is decreased and distortionsare increased.

Ortho-imaging is an approach that has been used in an attempt to addressthis problem. In general, ortho-imaging renders a composite image of atarget by compiling varying sub-images of the target. Typically, inaerial imaging applications, a digital imaging device that has a finiterange and resolution records images of fixed subsections of a targetarea sequentially. Those images are then aligned according to somesequence to render a composite of a target area.

Often, such rendering processes are very time-consuming and laborintensive. In many cases, those processes require iterative processingthat measurably degrades image quality and resolution—especially incases where thousands of sub-images are being rendered. In cases wherethe imaging data can be processed automatically, that data is oftenrepetitively transformed and sampled—reducing color fidelity and imagesharpness with each successive manipulation. If automated correction orbalancing systems are employed, such systems may be susceptible to imageanomalies (e.g., unusually bright or dark objects)—leading to over orunder-corrections and unreliable interpretations of image data. In caseswhere manual rendering of images is required or desired, time and laborcosts are immense.

There is, therefore, a need for an ortho-image rendering system thatprovides efficient and versatile imaging for very large FOVs andassociated data sets, while maintaining image quality, accuracy,positional accuracy and clarity. Additionally, automation algorithms areapplied extensively in every phase of the planning, collecting,navigating, and processing all related operations.

SUMMARY OF THE INVENTION

The present invention relates to remote data collection and processingsystem using a variety of sensors. The system may include computerconsole units that control vehicle and system operations in real-time.The system may also include global positioning systems that are linkedto and communicate with the computer consoles. Additionally, camerasand/or camera array assemblies can be employed for producing an image ofa target viewed through an aperture. The camera array assemblies arecommunicatively connected to the computer consoles. The camera arrayassembly has a mount housing, a first imaging sensor centrally coupledto the housing having a first focal axis passing through the aperture.The camera array assembly also has a second imaging sensor coupled tothe housing and offset from the first imaging sensor along an axis, thathas a second focal axis passing through the aperture and intersectingthe first focal axis within an intersection area. The camera arrayassembly has a third imaging sensor, coupled to the housing and offsetfrom the first imaging sensor along the axis, opposite the secondimaging sensor, that has a third focal axis passing through the apertureand intersecting the first focal axis within the intersection area. Anynumber of one-to-n cameras may be used in this manner, where “n” can beany odd or even number.

The system may also include an Attitude Measurement Unit (AMU) such asinertial, optical, or similar measurement units communicativelyconnected to the computer consoles and the camera array assemblies. TheAMU may determine the yaw, pitch, and/or roll of the aircraft at anyinstant in time and successive DGPS positions may be used to measure thevehicle heading with relation to geodesic north. The AMU data isintegrated with the precision DGPS data to produce a robust, real-timeAMU system. The system may further include a mosaicing module housedwithin the computer consoles. The mosaicing module includes a firstcomponent for performing initial processing on an input image. Themosaicing module also includes a second component for determininggeographical boundaries of an input image with the second componentbeing cooperatively engaged with the first component. The mosaicingmodule further includes a third component for mapping an input imageinto the composite image with accurate geographical position. The thirdcomponent being cooperatively engaged with the first and secondcomponents. A fourth component is also included in the mosaicing modulefor balancing color of the input images mapped into the composite image.The fourth component can be cooperatively engaged with the first, secondand third components. Additionally, the mosaicing module can include afifth component for blending borders between adjacent input imagesmapped into the composite image. The fifth component being cooperativelyengaged with the first, second, third and fourth components.

A sixth component, an optional forward oblique and/or optional rearoblique camera array system may be implemented that collects obliqueimage data and merges the image data with attitude and positionalmeasurements in order to create a digital elevation model usingstereographic techniques. Creation of which may be performed inreal-time onboard the vehicle or post processed later. This sixthcomponent works cooperatively with the other components. All componentsmay be mounted to a rigid platform for the purpose of providingco-registration of sensor data. Vibrations, turbulence, and other forcesmay act on the vehicle in such a way as to create errors in thealignment relationship between sensors. Utilization of common, rigidplatform mount for the sensors provides a significant advantage overother systems that do not use this co-registration architecture.

Further, the present invention may employ a certain degree of lateraloversampling to improve output quality and/or co-mounted, co-registeredoversampling to overcome physical pixel resolution limits.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the invention, and to show by way ofexample how the same may be carried into effect, reference is now madeto the detailed description of the invention along with the accompanyingfigures in which corresponding numerals in the different figures referto corresponding parts and in which:

FIG. 1 illustrates a vehicle based data collection and processing systemof the present invention;

FIG. 1A illustrates a portion of the vehicle based data collection andprocessing system of FIG. 1;

FIG. 1B illustrates a portion of the vehicle based data collection andprocessing system of FIG. 1;

FIG. 2 illustrates a vehicle based data collection and processing systemof FIG. 1 with the camera array assembly of the present invention shownin more detail;

FIG. 3 illustrates a camera array assembly in accordance with certainaspects of the present invention;

FIG. 4 illustrates one embodiment of an imaging pattern retrieved by thecamera array assembly of FIG. 1;

FIG. 5 depicts an imaging pattern illustrating certain aspects of thepresent invention;

FIG. 6 illustrates an image strip in accordance with the presentinvention;

FIG. 7 illustrates another embodiment of an image strip in accordancewith the present invention;

FIG. 8 illustrates one embodiment of an imaging process in accordancewith the present invention;

FIG. 9 illustrates diagrammatically how photos taken with the cameraarray assembly can be aligned to make an individual frame;

FIG. 10 is a block diagram of the processing logic according to certainembodiments of the present invention;

FIG. 11 is an illustration of lateral oversampling looking down from avehicle according to certain embodiments of the present invention;

FIG. 12 is an illustration of lateral oversampling looking down from avehicle according to certain embodiments of the present invention;

FIG. 13 is an illustration of flight line oversampling looking down froma vehicle according to certain embodiments of the present invention;

FIG. 14 is an illustration of flight line oversampling looking down froma vehicle according to certain embodiments of the present invention;

FIG. 15 is an illustration of progressive magnification looking downfrom a vehicle according to certain embodiments of the presentinvention;

FIG. 16 is an illustration of progressive magnification looking downfrom a vehicle according to certain embodiments of the presentinvention;

FIG. 17 is an illustration of progressive magnification looking downfrom a vehicle according to certain embodiments of the presentinvention;

FIG. 18 is a schematic of the system architecture according to certainembodiments of the present invention;

FIG. 19 is an illustration of lateral co-mounted, co-registeredoversampling in a sidelap sub-pixel area for a single camera arraylooking down from a vehicle according to certain embodiments of thepresent invention;

FIG. 20 is an illustration of lateral co-mounted, co-registeredoversampling in a sidelap sub-pixel area for two overlapping cameraarrays looking down from a vehicle according to certain embodiments ofthe present invention; and

FIG. 21 is an illustration of fore and lateral co-mounted, co-registeredoversampling in sidelap sub-pixel areas for two stereo camera arrayslooking down from a vehicle according to certain embodiments of thepresent invention.

DETAILED DESCRIPTION OF THE INVENTION

While the making and using of various embodiments of the presentinvention are discussed in detail below, it should be appreciated thatthe present invention provides many applicable inventive concepts, whichcan be embodied in a wide variety of specific contexts. The specificembodiments discussed herein are merely illustrative of specific ways tomake and use the invention and do not limit the scope of the invention.

A vehicle based data collection and processing system 100 of the presentinvention is shown in FIGS. 1, 1A, and 1B. Additional aspects andembodiments of the present invention are shown in FIGS. 2 and 18. System100 includes one or more computer consoles 102. The computer consolescontain one or more computers 104 for controlling both vehicle andsystem operations. Examples of the functions of the computer console arethe controlling digital color sensor systems that can be associated withthe data collection and processing system, providing the display data toa pilot, coordinating the satellite generated GPS pulse-per-second (PPS)event trigger (which may be 20 or more pulses per second), data logging,sensor control and adjustment, checking and alarming for error events,recording and indexing photos, storing and processing data, flightplanning capability that automates the navigation of the vehicle, data,and providing a real-time display of pertinent information. Acommunications interface between the control computer console and thevehicle autopilot control provides the ability to actually control theflight path of the vehicle in real-time. This results in a more precisecontrol of the vehicle's path than is possible by a human being. All ofthese functions can be accomplished by the use of various computerprograms that are synchronized to the GPS PPS signals and take intoaccount the various electrical latencies of the measurement devices. Inan embodiment, the computer is embedded within the sensor.

One or more differential global positioning systems 106 are incorporatedinto the system 100. The global positioning systems 106 are used tonavigate and determine precise flight paths during vehicle and systemoperations. To accomplish this, the global positioning systems 106 arecommunicatively linked to the computer console 102 such that theinformation from the global positioning systems 106 can be acquired andprocessed without flight interruption. Zero or more GPS units may belocated at known survey points in order to provide a record of eachsub-seconds' GPS satellite-based errors in order to be able to backcorrect the accuracy of the system 100. GPS and/or ground basedpositioning services may be used that eliminate the need for groundcontrol points altogether. This technique results in greatly improved,sub-second by sub-second positional accuracy of the data capturevehicle.

One or more AMUs 108 that provide real-time yaw, pitch, and rollinformation that is used to accurately determine the attitude of thevehicle at the instant of data capture are also communicatively linkedto the computer console 102. The present attitude measurement unit (AMU)(e.g., Applanix POS AV), uses three high performance fiber optic gyros,one gyro each for yaw, pitch, and roll measurement. AMUs from othermanufacturers, and AMUs that use other inertial measurement devices canbe used as well. Additionally, an AMU may be employed to determine theinstantaneous attitude of the vehicle and make the system more faulttolerant to statistical errors in AMU readings. Connected to the AMU canbe one or more multi-frequency DGPS receivers 110. The multi-frequencyDGPS receivers 110 can be integrated with the AMU's yaw, pitch, and rollattitude data in order to more accurately determine the location of theremote sensor platform in three dimensional space. Additionally, thedirection of geodesic North may be determined by the vector created bysuccessive DGPS positions, recorded in a synchronized manner with theGPS PPS signals.

One or more camera array assemblies 112 for producing an image of atarget viewed through an aperture are also communicatively connected tothe one or more computer consoles 102. The camera array assemblies 112,which will be described in greater detail below, provide the datacollection and processing system with the ability to capture highresolution, high precision progressive scan or line scan, color digitalphotography.

The system may also include DC power and conditioning equipment 114 tocondition DC power and to invert DC power to AC power in order toprovide electrical power for the system. The system may further includea navigational display 116, which graphically renders the position ofthe vehicle versus the flight plan for use by the pilot (either onboardor remote) of the vehicle to enable precision flight paths in horizontaland vertical planes. The system may also include an EMU module comprisedof LIDAR, SAR 118 or a forward and rear oblique camera array forcapturing three dimensional elevation/relief data. The EMU module 118can include a laser unit 120, an EMU control unit 122, and an EMUcontrol computer 124. Temperature controlling devices, such as solidstate cooling modules, can also be deployed as needed in order toprovide the proper thermal environment for the system.

The system also includes a mosaicing module, not depicted, housed withthe computer console 102. The mosaicing module, which will be describedin further detail below, provides the system the ability to gather dataacquired by the global positioning system 106, the AMU 108, and thecamera system 112 and process that data into useable orthomaps.

The system 100 also can include a self-locking flight path techniquethat provides the ability to micro-correct the positional accuracy ofadjacent flight paths in order to realize precision that exceeds thenative precision of the AMU and DGPS sensors alone.

A complete flight planning methodology is used to micro plan all aspectsof missions. The inputs are the various mission parameters(latitude/longitude, resolution, color, accuracy, etc.) and the outputsare detailed on-line digital maps and data files that are stored onboardthe data collection vehicle and used for real-time navigation andalarms. The ability to interface the flight planning data directly intothe autopilot is an additional integrated capability. A computer programmay be used that automatically controls the flight path, attitudeadjustments, graphical display, moving maps of the vehicle path, checksfor alarm conditions and corrective actions, notifies the pilot and/orcrew of overall system status, and provides for fail-safe operations andcontrols. Safe operations parameters may be constantly monitored andreported. Whereas the current system uses a manned crew, the system isdesigned to perform equally well in an unmanned vehicle.

FIG. 2 shows another depiction of the present invention. In FIG. 2, thecamera array assembly 112 is shown in more detail. As is shown, thecamera array assembly 112 allows for images to be acquired from the rearoblique, the forward obliques and the nadir positions. FIG. 3 describesin more detail a camera array assembly of the present invention. FIG. 3provides a camera array assembly 300 airborne over target 302 (e.g.,terrain). For illustrative purposes, the relative size of assembly 300,and the relative distance between it and terrain 302, are not depictedto scale in FIG. 3. The camera array assembly 300 comprises a housing304 within which imaging sensors 306, 308, 310, 312 and 314 are disposedalong a concave curvilinear axis 316. The radius of curvature of axis316 may vary or be altered dramatically, providing the ability to effectvery subtle or very drastic degrees of concavity in axis 316.Alternatively, axis 316 may be completely linear—having no curvature atall. The imaging sensors 306, 308, 310, 312 and 314 couple to thehousing 304, either directly or indirectly, by attachment members 318.Attachment members 318 may comprise a number of fixed or dynamic,permanent or temporary, connective apparatus. For example, theattachment members 318 may comprise simple welds, removable clampingdevices, or electro-mechanically controlled universal joints.

Additionally, the system 100 may have a real-time, onboard navigationsystem to provide a visual, bio-feedback display to the vehicle pilot,or remote display in the case of operations in an unmanned vehicle. Thepilot is able to adjust the position of the vehicle in real-time inorder to provide a more accurate flight path. The pilot may be onboardthe vehicle or remotely located and using the flight display to controlthe vehicle through a communication link.

The system 100 may also use highly fault-tolerant methods that have beendeveloped to provide a software inter-leaved disk storage methodologythat allows one or two hard drives to fail and still not lose targetdata that is stored on the drives. This software inter-leaved diskstorage methodology provides superior fault-tolerance and portabilityversus other, hardware methodologies, such as RAID-5.

The system 100 may also incorporate a methodology that has beendeveloped that allows for a short calibration step just before missiondata capture. The calibration methodology step adjusts the camerasettings, mainly exposure time, based on sampling the ambient lightintensity and setting near optimal values just before reaching theregion of interest. A moving average algorithm is then used to makesecond-by-second camera adjustments in order to deliver improved,consistent photo results. This improves the color processing of theorthomaps. Additionally, the calibration may be used to check or toestablish the exact spatial position of each sensor device (cameras,DPG, AMU, EMU, etc.). In this manner, changes that may happen in thespatial location of these devices may be accounted for and maintainoverall system precision metrics.

Additionally, the system 100 may incorporate a methodology that has beendeveloped that allows for calibrating the precision position andattitude of each sensor device (cameras, DPG, AMU, EMU, etc.) on thevehicle by flying over an area that contains multiple known, visible,highly accurate geographic positions. A program takes this data as inputand outputs the micro positional data that is then used to preciselyprocess the orthomaps.

As depicted in FIG. 3, housing 304 comprises a simple enclosure insideof which imaging sensors 306, 308, 310, 312 and 314 are disposed.Whereas FIG. 3 depicts a 5-camera array, the system works equally wellwhen utilizing any number of camera sensors from 1 to any number.Sensors 306 through 314 couple, via the attachment members 318, eithercollectively to a single transverse cross member, or individually tolateral cross members disposed between opposing walls of the housing304. In alternative embodiments, the housing 304 may itself compriseonly a supporting cross member of concave curvature to which the imagingsensors 306 through 314 couple, via members 318. In other embodiments,the housing 304 may comprise a hybrid combination of enclosure andsupporting cross member. The housing 304 further comprises an aperture320 formed in its surface, between the imaging sensors and target 302.Depending upon the specific type of host craft, the aperture 320 maycomprise only a void, or it may comprise a protective screen or windowto maintain environmental integrity within the housing 304. In the eventthat a protective transparent plate is used for any sensor, specialcoatings may be applied to the plate to improve the quality of thesensor data. Optionally, the aperture 320 may comprise a lens or otheroptical device to enhance or alter the nature of the images recorded bythe sensors. The aperture 320 is formed with a size and shape sufficientto provide the imaging sensors 306 through 314 proper lines of sight toa target region 322 on terrain 302.

The imaging sensors 306 through 314 are disposed within or along housing304 such that the focal axes of all sensors converge and intersect eachother within an intersection area bounded by the aperture 320. Dependingupon the type of image data being collected, the specific imagingsensors used, and other optics or equipment employed, it may benecessary or desirable to offset the intersection area or point ofconvergence above or below the aperture 320. The imaging sensors 306through 314 are separated from each other at angular intervals. Theexact angle of displacement between the imaging sensors may vary widelydepending upon the number of imaging sensors utilized and on the type ofimaging data being collected. The angular displacement between theimaging sensors may also be unequal, if required, so as to provide adesired image offset or alignment. Depending upon the number of imagingsensors utilized, and the particular configuration of the array, thefocal axes of all imaging sensors may intersect at exactly the samepoint, or may intersect at a plurality of points, all within closeproximity to each other and within the intersection area defined by theaperture 320.

As depicted in FIG. 3, the imaging sensor 310 is centrally disposedwithin the housing 304 along axis 316. The imaging sensor 310 has afocal axis 324, directed orthogonally from the housing 304 to align theline of sight of the imaging sensor with the image area 326 of theregion 322. The imaging sensor 308 is disposed within the housing 304along the axis 316, adjacent to the imaging sensor 310. The imagingsensor 308 is aligned such that its line of sight coincides with theimage area 328 of the region 322, and such that its focal axis 330converges with and intersects the axis 324 within the area bounded bythe aperture 320. The imaging sensor 312 is disposed within the housing304 adjacent to the imaging sensor 310, on the opposite side of the axis316 as the imaging sensor 308. The imaging sensor 312 is aligned suchthat its line of sight coincides with the image area 332 of the region322, and such that its focal axis 334 converges with and intersects axes324 and 330 within the area bounded by the aperture 320. The imagingsensor 306 is disposed within the housing 304 along the axis 316,adjacent to the sensor 308. The imaging sensor 306 is aligned such thatits line of sight coincides with the image area 336 of region 322, andsuch that its focal axis 338 converges with and intersects the otherfocal axes within the area bounded by aperture 320. The imaging sensor314 is disposed within housing 304 adjacent to sensor 312, on theopposite side of axis 316 as sensor 306. The imaging sensor 314 isaligned such that its line of sight coincides with image area 340 ofregion 322, and such that its focal axis 344 converges with andintersects the other focal axes within the area bounded by aperture 320.

The imaging sensors 306 through 314 may comprise a number of digitalimaging devices including, for example, individual area scan cameras,line scan cameras, infrared sensors, hyperspectral and/or seismicsensors. Each sensor may comprise an individual imaging device, or mayitself comprise an imaging array. The imaging sensors 306 through 314may all be of a homogenous nature, or may comprise a combination ofvaried imaging devices. For ease of reference, the imaging sensors 306through 314 are hereafter referred to as cameras 306 through 314,respectively.

In large-format film or digital cameras, lens distortion is typically asource of imaging problems. Each individual lens must be carefullycalibrated to determine precise distortion factors. In one embodiment ofthis invention, small-format digital cameras having lens angle widths of17 degrees or smaller are utilized. This alleviates noticeabledistortion efficiently and affordably.

Cameras 306 through 314 are alternately disposed within housing 304along axis 316 such that each camera's focal axis converges uponaperture 320, crosses focal axis 324, and aligns its field of view witha target area opposite its respective position in the array resulting ina “cross-eyed”, retinal relationship between the cameras and the imagingtarget(s). The camera array assembly 300 is configured such thatadjoining borders of image areas 326, 328, 332, 336 and 340 overlapslightly.

If the attachment members 318 are of a permanent and fixed nature (e.g.,welds), then the spatial relationship between the aperture 320, thecameras, and their lines of sight remain fixed as will the spatialrelationship between image areas 326, 328, 332, 336 and 340. Such aconfiguration may be desirable in, for example, a satellite surveillanceapplication where the camera array assembly 300 will remain at anessentially fixed distance from region 322. The position and alignmentof the cameras is set such that areas 326, 328, 332, 336 and 340 providefull imaging coverage of region 322. If the attachment members 318 areof a temporary or adjustable nature, however, it may be desirable toselectively adjust, either manually or by remote automation, theposition or alignment of the cameras so as to shift, narrow or widenareas 326, 328, 332, 336 and 340—thereby enhancing or altering thequality of images collected by the camera array assembly 300.

In an embodiment, multiple, i.e., at least two, rigid mount units areaffixed to the same rigid mount plate. The mount unit is any rigidstructure to which at least one imaging sensor may be affixed. The mountunit is preferably a housing, which encloses the imaging sensor, but maybe any rigid structure including a brace, tripod, or the like. For thepurposes of this disclosure, an imaging sensor means any device capableof receiving and processing active or passive radiometric energy, i.e.,light, sound, heat, gravity, and the like, from a target area. Inparticular, imaging sensors may include any number of digital cameras,including those that utilize a red-blue-green filter, a bushbroomfilter, or a hyperspectral filter, LIDAR sensors, infrared sensors,heat-sensing sensors, gravitometers and the like. Imagining sensors donot include attitude measuring sensors such as gyroscopes, GPS devices,and the like devices, which serve to orient the vehicle with the aid ofsatellite data and/or inertial data. Preferably, the multiple sensorsare different.

In the embodiment wherein the imaging sensor is a camera, LIDAR, or thelike imaging sensor, the mount unit preferably has an aperture throughwhich light and/or energy may pass. The mount plate is preferablyplaner, but may be non-planer. In the embodiment, wherein the imagingsensor is a camera, LIDAR, or the like imaging sensor, the mount platepreferably has aperture(s) in alignment with the aperture(s) of themount unit(s) through which light and/or energy may pass.

A rigid structure is one that flexes less than about 100^(th) of adegree, preferably less than about 1,000^(th) of a degree, morepreferably less than about 10,000^(th) of a degree while in use.Preferably, the rigid structure is one that flexes less than about100^(th) of a degree, preferably less than about 1,000^(th) of a degree,more preferably less than about 10,000^(th) of a degree while secured toan aircraft during normal, i.e., non-turbulent, flight. Objects arerigidly affixed to one another if during normal operation they flex fromeach other less than about 100^(th) of a degree, preferably less thanabout 1,000^(th) of a degree, more preferably less than about10,000^(th) of a degree.

Camera 310 is designated as the principal camera. The image plane 326 ofcamera 310 serves as a plane of reference. The orientations of the othercameras 306, 308, 312 and 314 are measured relative to the plane ofreference. The relative orientations of each camera are measured interms of the yaw, pitch and roll angles required to rotate the imageplane of the camera to become parallel to the plane of reference. Theorder of rotations is preferably yaw, pitch, and roll.

The imaging sensors affixed to the mount unit(s) may not be aligned inthe same plane. Instead, the angle of their mount relative to the mountangle of a first sensor affixed to the first mount unit, preferably theprinciple nadir camera of the first mount unit, may be offset.Accordingly, the imaging sensors may be co-registered to calibrate thephysical mount angle offset of each imaging sensor relative to eachother. In an embodiment, multiple, i.e., at least two, rigid mount unitsare affixed to the same rigid mount plate and are co-registered. In anembodiment, the cameras 306 through 314 are affixed to a rigid mountunit and co-registered. In this embodiment, the geometric centerpoint ofthe AMU, preferably a gyroscope, is determined using GPS and inertialdata. The physical position of the first sensor affixed to the firstmount unit, preferably the principle nadir camera of the first mountunit, is calculated relative to a reference point, preferably thegeometric centerpoint of the AMU. Likewise, the physical position of allremaining sensors within all mount units are calculated—directly orindirectly—relative to the same reference point.

The boresight angle of a sensor is defined as the angle from thegeometric center of that sensor to a reference plane. Preferably thereference plane is orthogonal to the target area. The boresight angle ofthe first sensor may be determined using the ground target points. Theboresight angles of subsequent sensors are preferably calculated withreference to the boresight angle of the first sensor. The sensors arepreferably calibrated using known ground targets, which are preferablyphoto-identifiable, and alternatively calibrated using a self-lockingflight path or any other method as disclosed in U.S. Publication No.2004/0054488A1, now U.S. Pat. No. 7,212,938B2, the disclosure of whichis hereby incorporated by reference in full.

The imaging sensor within the second mount unit may be any imagingsensor, and is preferably a LIDAR. Alternative, the second imagingsensor is a digital camera, or array of digital cameras. In anembodiment, the boresight angle of the sensor(s) affixed to the secondmount unit are calculated with reference to the boresight angle of thefirst sensor. The physical offset of the imaging sensor(s) within thesecond mount unit may be calibrated with reference to the boresightangle of the first sensor within the first mount unit.

In this manner, all of the sensors are calibrated at substantially thesame epoch, using the same GPS signal, the same ground target(s), andunder substantially the same atmospheric conditions. This substantiallyreduces compounded error realized when calibrating each sensorseparately, using different GPS signals, against different groundtargets, and under different atmospheric conditions.

Referring now to FIG. 4, images of areas 336, 328, 326, 332 and 340taken by cameras 306 through 314, respectively, are illustrated from anoverhead view. Again, because of the “cross-eyed” arrangement, the imageof area 336 is taken by camera 306, the image of area 340 is taken bycamera 314, and so on. In one embodiment of the present invention,images other than those taken by the center camera 310 take on atrapezoidal shape after perspective transformation. Cameras 306 through314 form an array along axis 316 that is, in most applications, pointeddown vertically. In an alternative embodiment, a second array ofcameras, configured similar the array of cameras 306 through 314, isaligned with respect to the first array of cameras to have an obliqueview providing a “heads-up” perspective. The angle of declination fromhorizontal of the heads-up camera array assembly may vary due to missionobjectives and parameters but angles of 25-45 degrees are typical. Otheralternative embodiments, varying the mounting of camera arrays, aresimilarly comprehended by the present invention. In all suchembodiments, the relative positions and attitudes of the cameras areprecisely measured and calibrated so as to facilitate image processingin accordance with the present invention.

In one embodiment of the present invention, an external mechanism (e.g.,a GPS timing signal) is used to trigger the cameras simultaneouslythereby capturing an array of input images. A mosaicing module thenrenders the individual input images from such an array into anortho-rectified compound image (or “mosaic”), without any visible seamsbetween the adjacent images. The mosaicing module performs a set oftasks comprising: determining the geographical boundaries and dimensionsof each input image; projecting each input image onto the mosaic withaccurate geographical positioning; balancing the color of the images inthe mosaic; and blending adjacent input images at their shared seams.The exact order of the tasks performed may vary, depending upon the sizeand nature of the input image data. In certain embodiments, themosaicing module performs only a single transformation to an originalinput image during mosaicing. That transformation can be represented bya 4×4 matrix. By combining multiple transformation matrices into asingle matrix, processing time is reduced and original input imagesharpness is retained.

During mapping of the input images to the mosaic, especially whenmosaicing is performed at high resolutions, pixels in the mosaic (i.e.,output pixels) may not be mapped to by any pixels in the input images(i.e., input pixels). Warped lines could potentially result as artifactsin the mosaic. Certain embodiments of the present invention overcomethis with a super-sampling system, where each input and output pixel isfurther divided into an n×m grid of sub-pixels. Transformation isperformed from sub-pixels to sub-pixels. The final value of an outputpixel is the average value of its sub-pixels for which there is acorresponding input sub-pixel. Larger n and m values produce mosaics ofhigher resolution, but do require extra processing time.

During its processing of image data, the mosaicing module may utilizethe following information: the spatial position (e.g., x, y, zcoordinates) of each camera's focal point at the time an input image iscaptured; the attitude (i.e., yaw, pitch, roll) of each camera's imageplane relative to the target region's ground plane at the time an inputimage was captured; each camera's fields of view (i.e., along track andcross track); and the Digital Terrain Model (DTM) of the area. Theattitude can be provided by the AMUs associated with the system. Digitalterrain models (DTMs) or Digital surface models (DSMs) can be createdfrom information obtained using a LIDAR module 118. LIDAR is similar tothe more familiar radar, and can be thought of as laser radar. In radar,radio waves are transmitted into the atmosphere that scatters some ofthe energy back to the radar's receiver. LIDAR also transmits andreceives electromagnetic radiation, but at a higher frequency since itoperates in the ultraviolet, visible and infrared region of theelectromagnetic spectrum. In operation, LIDAR transmits light out to atarget area. The transmitted light interacts with and is changed by thetarget area. Some of this light is reflected/scattered back to the LIDARinstrument where it can be analyzed. The change in the properties of thelight enables some property of the target area to be determined. Thetime for the light to travel out to the target area and back to LIDARdevice is used to determine the range to the target.

DTM and DSM data sets can also be captured from the camera arrayassembly. Traditional means of obtaining elevation data may also be usedsuch as stereographic techniques.

There are presently three basic types of LIDAR: Range finders,Differential Absorption LIDAR (DIAL) and Doppler LIDAR. Range finderLIDAR is the simplest LIDAR and is used to measure the distance from theLIDAR device to a solid or hard target. DIAL LIDAR is used to measurechemical concentrations (such as ozone, water vapor, pollutants) in theatmosphere. A DIAL LIDAR uses two different laser wavelengths that areselected so that one of the wavelengths is absorbed by the molecule ofinterest while the other wavelength is not. The difference in intensityof the two return signals can be used to deduce the concentration of themolecule being investigated. Doppler LIDAR is used to measure thevelocity of a target. When the light transmitted from the LIDAR hits atarget moving towards or away from the LIDAR, the wavelength of thelight reflected/scattered off the target will be changed slightly. Thisis known as a Doppler-shift and therefore Doppler LIDAR. If the targetis moving away from the LIDAR, the return light will have a longerwavelength (sometimes referred to as a red shift), if moving towards theLIDAR the return light will be at a shorter wavelength (blue shifted).The target can be either a hard target or an atmospheric target (e.g.microscopic dust and aerosol particles that are carried by the wind.

A camera's focal point is preferably used as a perspectivetransformation center. Its position in space may be determined, forexample, by a multi-frequency carrier phase post-processed GPS systemmounted on the host craft. The offsets, in three dimensions, of acamera's focal point are preferably carefully measured against thecenter of the GPS antenna. These offsets may be combined with theposition of the GPS antenna, and the orientation of the host craft, todetermine the exact position of the camera's focal point. The positionof the GPS antenna is preferably determined by processing of collectedGPS data against similar ground-based GPS antennas deployed at preciselysurveyed points.

One or more AMUs (e.g., the Applanix POS AV) are preferably mountedonboard for attitude determination. The attitude of the AMU referenceplane relative to the target region's ground plane is preferablymeasured and recorded at short intervals, with accuracy better thanone-hundredth of one degree. The attitude of the AMU reference plane maybe defined as the series of rotations that can be performed on the axesof this plane to make it parallel to the ground plane. The term “align”could also be used to describe this operation.

The attitude of center camera 310 (i.e. its image plane), relative tothe AMU, is preferably precisely calibrated. The attitude of each of theother cameras, relative to center camera 310, is preferably also becarefully calibrated. This dependent calibration is more efficient thandirectly calibrating each camera. When the camera array assembly 300 isremounted, only center camera 310 needs to be recalibrated. Effectively,a series of two transformations is applied to an input image from centercamera 310. First, the center camera's image plane is aligned to the AMUplane. Then, the AMU plane is aligned again to the ground plane. Thesetransformations, however, combine into a single operation by multiplyingtheir respective transformation matrices. For images from each of theother cameras, an additional transformation is first performed to alignit with the center camera's image plane.

The position of the focal point of center camera 310 may be determinedas described above. The x and y components of this position preferablydetermine the position of the mosaic's nadir point 400 on the ground.Field of view (FOV) angles of each camera are known, thus the dimensionsof each input image may be determined by the z component of thatcamera's focal point. An average elevation of the ground is preferablydetermined by computing the average elevation of points in the DTMs ofthe area, and then each input image is projected to an imaginaryhorizontal plane at this elevation. Relief displacement is thenpreferably applied using the DTMs of the area. The DTMs can be obtainedfrom many sources including: the USGS 30- or 10-meter DTMs available formost of the US; commercial DTMs; or DTMs obtained by a LIDAR or SAR EMUdevice mounted on the host craft that captures data concurrently withthe cameras.

Besides being geographically correctly placed, the resulting compoundimage also needs to have radiometric consistency throughout, and novisible seams at the joints between two adjacent images. The presentinvention provides a number of techniques for achieving this goal.

A characteristic of a conventional camera is the exposure time (i.e.,the time the shutter is open to collect light onto the image plane). Thelonger the exposure time, the lighter the resultant image becomes.Exposure time must adapt to changes in ambient lighting caused byconditions such as: cloud coverage; the angle and position of the sunrelative to the camera; and so forth. Optimal exposure time may alsodepend on a camera's orientation with respect to lighting sources (e.g.,cameras pointing towards a sunlit object typically receive more ambientlight than those pointing towards a shaded object). Exposure time isadjusted to keep the average intensity of an image within a certaindesired range. For example, in 24-bit color images each Red, Green andBlue component can have intensity values from 0 to 255. In mostinstances, however, it is desirable to keep the average intensity at amean value (i.e., 127).

In the present invention, an exposure control module controls exposuretime for each of the cameras or imaging sensors. It examines each inputimage and calculates average image intensity. Based on a moving average(i.e., average intensity of the last X number of images), the exposurecontrol module determines whether to increase or decrease exposure time.The module can use a longer running average to effect a slower reactionto changes in lighting conditions, with less susceptibility to unusuallydark or light images (e.g., asphalt roads or water). The exposurecontrol module controls exposure time for each camera separately.

In systems where cameras are mounted without forward-motion compensationmechanisms, there must be a maximum limit for exposure time. Settingexposure time to a value larger than the maximum may causemotion-induced blurriness. For example, assume cameras are mounted on anairplane traveling at 170 miles/hour (or about 3 inches/ms). Assumedesired pixel resolution is 6 inches. Forward motion during imagecapture should be limited to half a pixel size—which in this case equals3 inches. Thus, maximum exposure for example is 1 millisecond.

In controlling imaging quality, it is useful to be able to determine ifchanges in light intensity are caused either due to a change in ambientlight or due to the presence of unusually light or dark objects (e.g.,reflecting water body, metal roofs, asphalts, etc.). Certainapplications of this invention involve aerial photography orsurveillance. It is observed that aerial images of the ground usuallycontain plants and vegetation—which have more consistent reflectivitythan water bodies or man-made structures such as roads and buildings. Ofcourse, images of plants and vegetation are usually green-dominant(i.e., the green component is the greatest of the red, green and bluevalues). Therefore, intensity correlation can be made more accurate byfocusing on the green-dominant pixels.

The exposure control module computes the average intensity of an imageby selecting only green-dominant pixels. For example, if an image has 1million pixels and 300,000 are green-dominant, only those 300,000green-dominant pixels are included in the calculation of averageintensity. This results in an imaging process that is less susceptibleto biasing caused by man-made structures and water bodies, whose pixelsare usually not green-dominant. As previously noted, it is desirable tomaintain an intensity value of about 127. When intensity value is over127 (i.e., over-exposed), exposure time is reduced so that less light iscaptured. Similarly, when intensity value is under 127 (i.e.,under-exposed), exposure time is increased so that more light iscaptured. For example, consider a system flying over a target terrainarea having many white roofs, whose intensities are very high. Averageintensity for the images captured would tend to be high. In mostconventional systems, exposure time would by reduced in order tocompensate. In such an example, however, reducing exposure time is notproper, because the average intensity of the images has been biased bythe bright roofs. Reducing exposure time would result in images wherethe ground is darker than it should be. In contrast, if onlygreen-dominant pixels are processed in accordance with the presentinvention, then pixels representing the overly bright roofs do bias theaverage intensity and the exposure time is not changed.

Thus, the exposure control module reduces intensity differences betweeninput images. Nonetheless, further processing is provided to enhancetonal balance. There are a number of factors (e.g., lens physics,atmospheric conditions, spatial/positional relationships of imagingdevices) that cause an uneven reception of light from the image plane.More light is received in the center of a camera or sensor than at theedges.

The mosaicing module of the present invention addresses this with ananti-vignetting function, illustrated in reference now to FIG. 5. Anumber of focal columns 500, 502, 504, 506 and 508 converge from imageplane 509 and cross through focal point 510 as they range across imagingtarget area 512 (e.g., ground terrain). Columns 500 through 508 maycomprise individual resolution columns of a single camera or sensor, ormay represent the focal axes of a number of independent cameras orsensors. For reference purposes, column 504 serves as the axis and point513 at which column 504 intersects image plane 509 serves as a principalpoint. The exposure control module applies an anti-vignetting functionmultiplying the original intensity of an input pixel with acolumn-dependent anti-vignetting factor. Because the receiving surfaceis represented as a plane with a coordinate system, each column willhave a number of resolution rows (not shown). This relationship may beexpressed, for a pixel p at column x and row y, as follows:<adjusted intensity>=<original intensity>*ƒ(x);where ƒ(x) is a function of the form:ƒ(x)=cos(off-axis angle)**4.The off-axis angle 514 is: zero for center column 504; larger forcolumns 502 and 506; and larger still for columns 500 and 508. Theoverall field of view angle 516 (FOVx angle) is depicted between columns504 and 508.

The function ƒ(x) can be approximated by a number of line segmentsbetween columns. For a point falling within a line segment between anygiven columns c1 and c2, an adjustment factor is computed as follows:<adjustment factor for c>=ƒ(c1)+[ƒ(c2)−ƒ(c1)*(c−c1)/(c2−c1)];

where ƒ(c1) and ƒ(c2) are the ƒ function values of the off-axis anglesat column c1 and c2, respectively.

Each set of input images needs to be stitched into a mosaic image. Eventhough the exposure control module regulates the amount of light eachcamera or sensor receives, the resulting input images may still differin intensity. The present invention provides an intensity-balancingmodule that compares overlapping area between adjacent input images, tofurther balance the relative intensities. Because adjoining input imagesare taken simultaneously, the overlapping areas should, in theory, haveidentical intensity in both input images. However, due to variousfactors, the intensity values are usually not the same. Some suchfactors causing intensity difference could include, for example, theexposure control module being biased by unusually bright or dark objectspresent in the field of view of only a particular camera, or theboresight angles of cameras being different (i.e., cameras that are moreslanted receive less light than those more vertical).

To balance two adjacent images, one is chosen as the reference image andthe other is the secondary image. A correlation vector (fR, fG, FB) isdetermined using, for example, the following process. Let V be a 3×1vector representing the values (R, G and B) of a pixel:

$V = {\begin{matrix}R \\G \\B\end{matrix}.}$A correlation matrix C may be derived as:

${C = \begin{matrix}{FR} & 0 & 0 \\0 & {FG} & 0 \\0 & 0 & {FB}\end{matrix}};$where FR=AvgIr/AvgIn; AvgIr=Red average intensity of overlapped regionin reference image; AvgIn=Red average intensity of overlapped region innew image; and FG and FB are similarly derived.

The correlation matrix scales pixel values of the secondary image sothat the average intensity of the overlapping area of the secondaryimage becomes identical to the average intensity of the overlapping areaof the reference image. The second image can be balanced to thereference image by multiplying its pixel values by the correlationmatrix.

Thus, in one embodiment of a balancing process according to the presentinvention, a center image is considered the reference image. Thereference image is first copied to the compound image (or mosaic).Overlapping areas between the reference image and an adjoining image(e.g., the near left image) are correlated to compute a balancingcorrelation matrix (BCM). The BCM will be multiplied with vectorsrepresenting pixels of the adjoining image to make the intensity of theoverlapping area identical in both images. One embodiment of thisrelationship may be expressed as:Let I(center)=Average intensity of overlapping area in center image;I(adjoining)=Average intensity of overlap in adjoining image; thenBalancing factor=I(center)/I(adjoining).

The balancing factor for each color channel (i.e., red, green and blue)is independently computed. These three values form the BCM. Thenow-balanced adjoining image is copied to the mosaic. Smoothtransitioning at the border of the copied image is providing by“feathering” with a mask. This mask has the same dimension as theadjoining image and comprises a number of elements. Each element in themask indicates the weight of the corresponding adjoining image pixel inthe mosaic. The weight is zero for pixels at the boundary (i.e. theoutput value is taken from the reference image), and increases graduallyin the direction of the adjoining image until it becomes unity—after achosen blending width has been reached. Beyond the blending area, themosaic will be entirely determined by the pixels of the adjoining image.Similarly, the overlaps between all the other constituent input imagesare analyzed and processed to compute the correlation vectors and tobalance the intensities of the images.

A correlation matrix is determined using, for example, the followingprocess with reference to FIG. 6. FIG. 6 depicts a strip 600 beingformed in accordance with the present invention. A base mosaic 602 and anew mosaic 604, added along path (or track) 606, overlap each other inregion 608. Let V be a vector that represents the R, G and B values of apixel:

$V = \begin{matrix}R \\G \\B\end{matrix}$Let h be the transition width of region 608, and y be the along-track606 distance from the boundary 610 of the overlapped region to a pointA, whose pixel values are represented by V. Let C be the correlationmatrix:

$C = \begin{matrix}{FR} & 0 & 0 \\0 & {FG} & 0 \\0 & 0 & {FB}\end{matrix}$The balanced value of V, called V′ is:V′=[y/h·I+(1−y/h)·C]×V, for 0<y<h;V′=V, for y>=h;

Where I is the identity matrix

$I = {\begin{matrix}1 & 0 & 0 \\0 & 1 & 0 \\0 & 0 & 1\end{matrix}.}$Note that the “feathering” technique is also used in combination withthe gradient to minimize seam visibility.

When mosaics are long, differences in intensity at the overlap maychange from one end of the mosaic to the other. Computing a singlecorrelation vector to avoid creating visible seams may not be possible.The mosaic can be divided into a number of segments corresponding to theposition of the original input images that make up the mosaic. Theprocess described above is applied to each segment separately to providebetter local color consistency.

Under this refined algorithm, pixels at the border of two segments maycreate vertical seams (assuming north-south flight lines). To avoid thisproblem, balancing factors for pixels in this area have to be“transitioned” from that of one segment to the other. This is explainednow with reference to FIG. 7.

FIG. 7 depicts a strip 700 being formed in accordance with the presentinvention. A base mosaic 702 and a new segment 704 overlap in area 706.Mosaic 702 and another new segment 708 overlap in area 710. Segments 704and 708 overlap in area 712, and areas 706, 710 and 712 all overlap andcoincide at area 714. For explanation purposes, point 716 serves as anorigin for y-axis 718 and x-axis 720. Movement along y-axis 718represents movement along the flight path of the imaging system. Point716 is located at the lower left of area 714.

According to the present invention, the dimensions of a strip aredetermined by the minimum and maximum x and y values of the constituentmosaics. An output strip is initialized to a background color. A firstmosaic is transferred to the strip. The next mosaic (along the flightpath) is processed next. Intensity values of the overlapping areas ofthe new mosaic and the first mosaic are correlated, separately for eachcolor channel. The new mosaic is divided into a number of segmentscorresponding to the original input images that made up the mosaic. Amask matrix, comprising a number of mask elements, is created for thenew mosaic. A mask element contains the correlation matrix for acorresponding pixel in the new mosaic. All elements in the mask areinitialized to unity. The size of the mask can be limited to just thetransition area of the new mosaic. The correlation matrix is calculatedfor the center segment. The mask area corresponding to the centersegment is processed. The values of the elements at the edge of theoverlap area are set to the correlation vector. Then, gradually movingaway from the first mosaic along the strip, the components of thecorrelation matrix are either increased or decreased (whether they areless or more than unity, respectively) until they become unity at apredetermined transition distance. The area of the mask corresponding toa segment adjoining the center segment is then processed similarly.However, the area 714 formed by the first mosaic and the center andadjoining segments of the new image requires special treatment. Becausethe correlation matrix for the adjoining segment may not be identical tothat of the center segment, a seam may appear at the border of the twosegments in the overlap area 714 with the first mosaic. Therefore, thecorner is influenced by the correlation matrices from both segments. Fora mask cell A at distance x to the border with the center segment anddistance y to the overlap edge, its correlation matrix is thedistance-weighted average of the two segments, evaluated as follows:

For pixel A(x, y) in area 714 at distance x to the border with thecenter segment, its balanced values are computed as thedistance-weighted averages of the values computed using the twosegments;

V1 is the balanced RGB vector based on segment 704;

V2 is the balanced RGB vector based on segment 708;

V′ is the combined (final) balanced RGB vectorV′=((d−x)/d)·V1+(x/d)·V2;Where

-   -   x-axis is the line going through bottom of overlapped region;    -   y-axis is the line going through the left side of the overlapped        region between segments 704 and 708;    -   h is the transition width; and    -   d is the width of the overlapped region between segments 704 and        708.        The mask areas corresponding to other adjoining segments are        computed similarly.

Further according to the present invention, a color fidelity (i.e.,white-balance) filter is applied. This multiplies R and B componentswith a determinable factor to enhance color fidelity. The factor may bedetermined by calibrating the cameras and lenses. The color fidelityfilter ensures that the colors in an image retain their fidelity, asperceived directly by the human eye. Within the image capture apparatus,the Red, Green and Blue light receiving elements may have differentsensitivities to the color they are supposed to capture. A“while-balance” process is applied—where image of a white object iscaptured. Theoretically, pixels in the image of that white object shouldhave equivalent R, G and B values. In reality, however, due to differentsensitivities and other factors, the average color values for each R, Gand B may be avgR, avgG and avgB, respectively. To equalize the colorcomponents, the R, G and B values of the pixels are multiplied by thefollowing ratios:

R values are multiplied by the ratio avgG/avgR; and

B values are multiplied by the ratio avgG/avgB.

The end result is that the image of the white object is set to haveequal R G B components.

In most applications, a strip usually covers a large area of non-watersurface. Thus, average intensity for the strip is unlikely to be skewedby anomalies such as highly reflecting surfaces. The present inventionprovides an intensity normalization module that normalizes the averageintensity of each strip so that the mean and standard deviation are of adesired value. For example, a mean of 127 is the norm in photogrammetry.A standard deviation of 51 helps to spread the intensity value over anoptimal range for visual perception of image features. Each strip mayhave been taken in different lighting conditions and, therefore, mayhave different imaging data profiles (i.e., mean intensity and standarddeviation). This module normalizes the strips, such that all have thesame mean and standard deviation. This enables the strips to be stitchedtogether without visible seams.

This intensity normalization comprises a computation of the meanintensity for each channel R, G and B, and for all channels. The overallstandard deviation is then computed. Each R, G and B value of each pixelis transformed to the new mean and standard deviation:new value=new mean+(old value−old mean)*(new std/old std).

Next, multiple adjacent strips are combined to produce tiled mosaics foran area of interest. Finished tiles can correspond to the USGS quads orquarter-quads. Stitching strips into mosaics is similar to stitchingmosaics together to generate strips, with strips now taking the role ofthe mosaics. At the seam line between two strips, problems may arise ifthe line crosses elevated structures such as buildings, bridges, etc.This classic problem in photogrammetry arises from the parallax causedby the same object being looked at from two different perspectives.During imaging of a building, for example, one strip may present a viewfrom one side of the building while another strip presents a view fromanother side of the building. After the images are stitched together,the resulting mosaic may look like a tepee. In order to address this, aterrain-guided mosaicing process may be implemented to guide theplacement of a seam line. For example, LIDAR or DEM data collected with,or analyzed from, image data may be processed to determine theconfiguration and shaping of images as they are mosaiced together. Thus,in some mosaiced images, a seam line may not be a straight line—insteadcomprising a seam line that shifts back and forth to snake throughelevated structures.

Referring now to FIG. 8, one embodiment of an imaging process 800 isillustrated in accordance with the present invention as described above.Process 800 begins with a series 802 of one, or more, raw collectedimages. Images 802 are then processed through a white-balancing process804, transforming them into a series of intermediate images. Series 802is then processed through anti-vignetting function 806 beforeprogressing to the orthorectification process 808. As previously noted,orthorectification may rely on position and attitude data 810 from theimaging sensor system or platform, and on DTM data 812. DTM data 812 maybe developed from position data 810 and from, for example, USGS DTM data814 or LIDAR data 816. Series 802 is now orthorectified and processingcontinues with color balancing 818. After color balancing, series 802 isconverted by mosaicing module 820 into compound image 822. Module 820performs the mosaicing and feathering processes during this conversion.Now, one or more compound images 822 are further combined in step 824,by mosaicing with a gradient and feathering, into image strip 826. Imagestrips are processed through intensity normalization 828. The nownormalized strips 828 are mosaiced together in step 830, again bymosaicing with a gradient and feathering, rendering a finishing tiledmosaic 832. The mosaicing performed in step 830 may comprise aterrain-guided mosaicing, relying on DTM data 812 or LIDAR data 816.

FIG. 9 illustrates diagrammatically how photos taken with the cameraarray assembly may be aligned to make an individual frame. Thisembodiment shows a photo patter illustration looking down from avehicle, using data ortho-rectified from five cameras.

FIG. 10 is a block diagram of the processing logic according to certainembodiments of the present invention. As shown in block diagram 1000,the processing logic accepts one or more inputs, which may includeelevation measurements 1002, attitude measurements 1004 and/or photo andsensor imagery 1006. Certain inputs may be passed through an initialprocessing step prior to analysis, as is shown in block 1008, whereinthe attitude measurements are combined with data from ground controlpoints. Elevation measurements 1002 and attitude measurements 1004 maybe combined to generate processed elevation data 1010. Processedelevation data 1010 may then be used to generate elevation DEM 1014 andDTM 1016. Similarly, attitude measurements 1006 may be combined withphoto and sensor imagery 1006 to generate georeferenced images 1012,which then undergo image processing 1018, which may include colorbalancing and gradient filtering.

Depending on the data set to be used (1020), either DTM 1016 or a USGSDEM 1022 is combined with processed images 1018 to generateorthorectified imagery 1024. Orthorectified imagery 1024 then feeds intoself-locking flightlines 1026. Balancing projection mosaicing 1028 thenfollows, to generate final photo output 1030.

The present invention may employ a certain degree of lateraloversampling to improve output quality. FIG. 11 is an illustration of alateral oversampling pattern 1100 looking down from a vehicle accordingto certain embodiments of the present invention showing minimal lateraloversampling. In this illustration, the central nadir region 1102assigned to the center camera overlaps only slightly with the left nadirregion 1104 and right nadir region 1106, so that overlap is minimized.FIG. 12 is an illustration of a lateral oversampling pattern 1200looking down from a vehicle according to certain embodiments of thepresent invention showing a greater degree of lateral oversampling. Inthis illustration, the central nadir region 1202 shows a high degree ofoverlap with left nadir region 1204 and right nadir region 1206.

In addition to the use of lateral oversampling as shown in FIGS. 11 and12, the present invention may employ flight line oversampling as well.FIG. 13 is an illustration of a flight line oversampling pattern 1300looking down from a vehicle according to certain embodiments of thepresent invention showing a certain degree of flight line oversamplingbut minimal lateral oversampling. Central nadir regions 1302 and 1304are overlapped to one another along the flight line, but do not overlaplaterally with left nadir regions 1306 and 1308 or with right nadirregions 1310 and 1312.

FIG. 14 is an illustration of flight line oversampling looking down froma vehicle according to certain embodiments of the present inventionshowing significant flight line oversampling as well as significantlateral oversampling. It can be seen that each of the central nadirregions 1402 through 1406 are significantly overlapped with one anotheras well as with left nadir regions 1408 through 1412 and right nadirregions 1414 through 1418. Left nadir regions 1408 through 1412 areoverlapped with one another, as are right nadir regions 1414 through1418. Accordingly, each point on the surface is sampled at least twice,and in some cases as many as four times. This technique uses the factthat in the area of an image that is covered twice, or more, bydifferent camera sensors, a doubling of the image resolution is possiblein both the lateral (across path) and flight line (along path)directions for an overall quadrupling of the resolution. In practice,the improvement in image/sensor resolution is somewhat less than doubledin each of the dimensions, approximately 40% in each dimension, or1.4×1.4=˜2 times. This is due to the statistical variations of thesub-pixel alignment/orientation. In effect, the pixel grid is rarelyexactly equidistant from the overlaid pixel grid. If extremely preciselateral camera sensor alignments were made at the sub-pixel level, aquadrupling of image resolution could be realized.

FIG. 15 is an illustration of a progressive magnification pattern 1500looking down from a vehicle according to certain embodiments of thepresent invention. Central nadir region 1502 is bounded on its left andright edges by inner left nadir region 1504 and inner right nadir region1506, respectively. Inner left nadir region 1504 is bounded on its leftedge by outer left nadir region 1508, while inner right nadir region1506 is bounded on its right edge by outer right nadir region 1510. Notethat these regions exhibit a minimal degree of overlap and oversamplingfrom one to another.

FIG. 16 is an illustration of a progressive magnification pattern 1600looking down from a vehicle according to certain embodiments of thepresent invention. Central nadir region 1602 is bounded on its left andright edges by inner left nadir region 1604 and inner right nadir region1606, respectively. Inner left nadir region 1604 is bounded on its leftedge by outer left nadir region 1608, while inner right nadir region1606 is bounded on its right edge by outer right nadir region 1610. Notethat, as above, these regions exhibit a minimal degree of overlap andoversampling from one to another. Within each of the nadir regions 1604through 1610, there is a central image region 1614 through 1620 shownshaded in grey.

FIG. 17 is an illustration of a progressive magnification pattern 1700looking down from a vehicle according to certain embodiments of thepresent invention. In the center of pattern 1700, a left inner nadirregion 1702 and a right inner nadir region 1704 overlap in the center. Aleft intermediate nadir region 1706 and a right intermediate nadirregion 1708 are disposed partly outside of regions 1702 and 1704,respectively, each sharing an overlapping area with the respectiveadjacent area by approximately 50%. An outer left nadir region 1710 andan outer right nadir region 1712 are disposed partly outside of regions1706 and 1708, respectively, each sharing an overlapping area with therespective adjacent area by approximately 50%. A central image region1714 is disposed in the center of pattern 1700, comprised of the centralportions of nadir regions 1702 through 1712.

FIG. 18 depicts a schematic of the architecture of a system 1800according to certain embodiments of the present invention. System 1800may include one or more GPS satellites 1802 and one or more SATCOMsatellites 1804. One or more GPS location systems 1806 may also beincluded, operably connected to one or more modules 1808 collectingLIDAR, GPS and/or X, Y, Z location data and feeding such information toone or more data capture system applications 1812. One or more datacapture system applications 1812 may also receive spectral data from acamera array 1822. A DGPS 1810 may communicate with one or more SATCOMsatellites 1804 via a wireless communications link 1826. One or moreSATCOM satellites 1804 may, in turn, communicate with one or more datacapture system applications 1812.

One or more data capture system applications 1812 may interface with anautopilot 1816, an SSD and/or a RealTime StitchG system 1820, which mayalso interact with one another. SSD 1814 may be operably connected toRealTime DEM 1818. Finally, RealTime DEM 1818 and RealTime StitchG 1820may be connected to a storage device, such as disk array 1824.

The present invention may employ a certain degree of co-mounted,co-registered oversampling to overcome physical pixel resolution limits.FIG. 19 is an illustration of a lateral co-mounted, co-registeredoversampling configuration 1900 for a single camera array 112 lookingdown from a vehicle according to certain embodiments of the presentinvention showing minimal lateral oversampling. The cameras overlap afew degrees in the vertical sidelap area 1904 and 1908. Whereas FIG. 19depicts a 3-camera array, these subpixel calibration techniques workequally well when utilizing any number of camera sensors from 2 to anynumber of cameras being calibrated.

Similar to the imaging sensors in FIGS. 3 and 4, the camera sensors maybe co-registered to calibrate the physical mount angle offset of eachsensor relative to each other and/or to the nadir camera. This providesan initial, “close” calibration. These initial calibration parametersmay be entered into an onboard computer system 104 in the system 100,and updated during flight using oversampling techniques.

Referring now to FIG. 19, the rectangles labeled A, B, and C representimage areas 1902, 1906 and 1910 from a 3-camera array C-B-A (not shown).Images of areas 1902, 1906 and 1910 taken by cameras A through C (notshown), respectively, are illustrated from an overhead view. Again,similar to FIGS. 3 and 4, because of the “cross-eyed” arrangement, theimage of area 1902 is taken by right camera A, the image of area 1906 istaken by center/nadir camera B, and the image of area 1910 is taken byleft camera C. Cameras A through C form an array (not shown) that is, inmost applications, pointed down vertically.

In FIG. 19, the hatched areas labeled A/B and B/C sidelaps representimage overlap areas 1904 and 1908, respectively. The left image overlaparea 1904 is where right camera A overlaps with the center/nadir cameraB, and the right image overlap area 1908 is where the left camera Coverlaps with the center/nadir camera B. In these sidelap areas 1904 and1908, the camera sensor grid bisects each pixel in the overlap areas1904 and 1908, which effectively quadruples the image resolution inthese areas 1904 and 1908 via the mechanism of co-mounted, co-registeredoversampling. In effect, the improvement in image/sensor resolution isdoubled in each dimension, or 2×2=4 times. This quadrupling of the imageresolution also quadruples the alignment precision between adjacentcameras.

Further, this quadrupling of alignment precision between adjacentcameras improves the systems 100 alignment precision for all sensorsaffixed to a rigid mount plate. The cameras and sensors are affixed to arigid mount unit, which is affixed to the rigid mount plate, asdiscussed above. In particular, when the angular alignment of adjacentcameras affixed to the rigid mount unit is improved, the angularalignment of the other sensors is also enhanced. This enhancement ofalignment precision for the other sensors affixed to the rigid mountplate also improves the image resolution for those sensors.

A lateral co-mounted, co-registered oversampling configuration 2000 fortwo overlapping camera arrays 112 is illustrated in FIG. 20. Inparticular, FIG. 20 is an illustration of a lateral co-mounted,co-registered oversampling configuration 2000 for two overlapping cameraarrays 112 looking down from a vehicle according to certain embodimentsof the present invention showing maximum lateral oversampling. Theadjacent cameras overlap a few degrees in the vertical sidelap areas2006, 2008, 2014 and 2016, and the corresponding cameras overlapcompletely in the image areas 2002, 2010, 2018 and 2004, 2012, 2020.Whereas FIG. 20 depicts two 3-camera arrays, these subpixel calibrationtechniques work equally well when utilizing two overlapping cameraarrays with any number of camera sensors from 2 to any number of camerasbeing calibrated.

Similar to the imaging sensors in FIGS. 3 and 4, the camera sensors maybe co-registered to calibrate the physical mount angle offset of eachsensor relative to each other and/or to the nadir camera. In thisembodiment, multiple, i.e., at least two, rigid mount units are affixedto a rigid mount plate and are co-registered. This provides an initial,“close” calibration. These initial calibration parameters may be enteredinto an onboard computer system 104 in the system 100, and updatedduring flight.

Referring now to FIG. 20, the rectangles labeled A, B, and C representimage areas 2002, 2010, 2018, and 2004, 2012, 2020 from two overlapping3-camera arrays C-B-A (not shown), respectively. Images of areas 2002,2010, 2018, and 2004, 2012, 2020 taken by cameras A through C (notshown) and overlapping cameras A′ through C′ (not shown), respectively,are illustrated from an overhead view. Again, similar to FIGS. 3 and 4,because of the “cross-eyed” arrangement, the image of area 2002 is takenby right camera A, the image of area 2010 is taken by center/nadircamera B, and the image of area 2018 is taken by left camera C. Further,the image of area 2004 is taken by right camera A′, the image of area2012 is taken by center camera B′, and the image of area 2020 is takenby left camera C′. Cameras A through C and overlapping cameras A′through C′ form arrays (not shown) that are, in most applications,pointed down vertically.

In FIG. 20, the hatched areas labeled A/B and B/C sidelaps represent twooverlapping image overlap areas 2006, 2008 and 2014, 2016, respectively.The left image overlap areas 2006, 2008 is where right camera A overlapswith the center/nadir camera B, and where right camera A′ overlaps withthe center camera B′, respectively. The right image overlap areas 2014and 2016 is where the left camera C overlaps with the center/nadircamera B, and where the left camera C′ overlaps with the center cameraB′. In these sidelap areas 2006, 2008 and 2014, 2016, respectively, thecamera sensor grid bisects each pixel in the overlap areas 2006, 2008and 2014, 2016, which effectively quadruples the image resolution inthese areas 2006, 2008 and 2014, 2016 via the mechanism of co-mounted,co-registered oversampling. In effect, the improvement in image/sensorresolution is doubled in each dimension, or 2×2=4 times. Thisquadrupling of the image resolution quadruples the alignment precisionbetween adjacent cameras, as discussed above.

By having two overlapping camera arrays, the image resolution iseffectively quadrupled again for the overlapping sidelap overlap areas2006, 2008 and 2014, 2016. This results in an astounding overall 64times improvement in system 100 calibration and camera alignment.

In the overlapping sidelap areas 2006 and 2008, the overlapping camerasensor grids bisects each pixel in the sidelap areas 2006 and 2008,which effectively quadruples the image resolution in these areas 2006and 2008 via the mechanism of co-mounted, co-registered oversampling.Similarly, in the overlapping sidelap areas 2014 and 2016, theoverlapping camera sensor grids bisects each pixel in the sidelap areas2014 and 2016, which effectively quadruples the image resolution inthese areas 2014 and 2016. In effect, the improvement in image/sensorresolution is again doubled in each dimension, or 2×2×2×2×2×2=64 times.This overall 64 times improvement of the image resolution also enhancesalignment precision by 64 times between adjacent cameras.

This 64 times improvement of alignment precision between adjacent andcorresponding cameras enhances the systems 100 alignment precision forall sensors affixed to a rigid mount plate. Cameras A through C and,optionally, other sensors are affixed to a first rigid mount unit andcameras A′ through C′ and, optionally, other sensors are affixed to asecond rigid mount unit, which are each affixed to a rigid mount plate.In particular, when the angular alignment of adjacent and/orcorresponding cameras affixed to the first and/or second rigid mountunits is improved, the angular alignment of the other sensors is alsoenhanced. This enhancement of alignment precision for the other sensorsaffixed to the rigid mount plate also improves the image resolution forthose sensors.

By having two overlapping camera arrays, the image resolution iseffectively quadrupled for the entire image, not just for the A/B andB/C sidelap overlap areas. Referring now to FIG. 20, the overlappinggrid detail labeled “OVERLAPPING GRID 4X” represents overlapping areas2022 and 2024 in right images areas 2018 and 2020, respectively. In theoverlapping areas 2022 and 2024, the overlapping camera sensor gridsbisects each pixel in the overlapping areas 2022 and 2024, whicheffectively quadruples the image resolution in these areas 2022 and 2024via the mechanism of co-mounted, co-registered oversampling. In effect,the improvement in image resolution is doubled in each dimension, or2×2=4 times.

In a preferred embodiment, one camera array is monochrome, and anothercamera array is red-green-blue. Even though each array covers differentcolor bands, simple image processing techniques are used so that allcolor bands realize the benefit of this increased resolution. Anotheradvantage provided by these techniques is that, in the case where onecamera array is red-green-blue and the other, overlapping camera arrayis an infrared or near infrared (or some other bandwidth), which resultsin a superior multi-spectral image.

Accordingly, all of the improvements (i.e., 4 times) identified for theembodiment of FIG. 19 discussed above apply to the embodiment of FIG.20, however, additional significant enhancements (i.e., 64 times) to thesystems 100 calibration precision and overall image resolution may berealized through the two overlapping camera arrays.

FIG. 21 is an illustration of a fore and lateral co-mounted,co-registered oversampling configuration 2100 for two camera arrays 112looking down from a vehicle according to certain embodiments of thepresent invention. In particular, FIG. 21 is an illustration of a foreand lateral co-mounted, co-registered oversampling configuration 2100for two overlapping camera arrays 112 looking down from a vehicleaccording to certain embodiments of the present invention showingminimal fore and minimal lateral oversampling. The adjacent camerasoverlap a few degrees in the vertical sidelap areas 2104, 2108, 2124 and2128, and the corresponding cameras overlap a few degrees along thehorizontal forelap areas 2112, 2116 and 2120. Whereas FIG. 21 depictstwo 3-camera arrays, these subpixel calibration techniques work equallywell when utilizing two overlapping camera arrays with any number ofcamera sensors from 2 to any number of cameras being calibrated.

Similar to the imaging sensors in FIGS. 3 and 4, the camera sensors maybe co-registered to calibrate the physical mount angle offset of eachsensor relative to each other and/or to the nadir camera. In thisembodiment, multiple, i.e., at least two, rigid mount units are affixedto a rigid mount plate and are co-registered. This provides an initial,“close” calibration. These initial calibration parameters may be enteredinto an onboard computer system 104 in the system 100, and updatedduring flight.

Referring now to FIG. 21, the rectangles labeled A, B, and C representimage areas 2102, 2106 and 2110 from a 3-camera array C-B-A (not shown),and the rectangles D, E, and F represent image areas 2122, 2126 and 2130from a 3-camera array F-E-D (not shown), respectively. Images of areas2102, 2106 and 2110 taken by cameras A through C (not shown), and imagesof areas 2122, 2126 and 2130 taken by cameras D through F (not shown),respectively, are illustrated from an overhead view. Again, similar toFIGS. 3 and 4, because of the “cross-eyed” arrangement, the rear, leftimage of area 2102 is taken by rear, right camera A, the rear, centerimage of area 2106 is taken by rear, center/nadir camera B, and therear, right image of area 2110 is taken by rear, left camera C. Further,the forward, left image of area 2122 is taken by forward, right cameraD, the forward, center image of area 2126 is taken by forward, centercamera E, and the forward, right image of area 2020 is taken by forward,left camera F. Cameras A through C and overlapping cameras D through Fform arrays (not shown) that are, in most applications, pointed downvertically.

In FIG. 21, the vertical hatched areas represent four image overlapareas 2104, 2108, 2124 and 2128. The rear, left image overlap area 2104is where rear, right camera A overlaps with the center/nadir camera B,and the rear, right image overlap area 2108 is where rear, left camera Coverlaps with the center/nadir camera B. The forward, left image overlaparea 2124 is where forward, right camera D overlaps with thecenter/nadir camera E, and the forward, right image overlap area 2128 iswhere forward, left camera F overlaps with the center camera E.

Referring now to FIG. 21, the overlapping grid detail labeled “SIDELAPAREA 4:1” represents overlapping sidelap overlap areas 2104, 2108 and2124, 2128. In these sidelap overlap areas 2104, 2108, 2124 and 2128,the camera sensor grid bisects each pixel in the overlap areas 2104,2108, 2124 and 2128, which effectively quadruples the image resolutionin these areas 2104, 2108, 2124 and 2128 via the mechanism ofco-mounted, co-registered oversampling. In effect, the improvement inimage/sensor resolution is doubled in each dimension, or 2×2=4 times.This quadrupling of the image resolution quadruples the alignmentprecision between adjacent cameras, as discussed above.

This quadrupling of alignment precision between adjacent camerasimproves the systems 100 alignment precision for all sensors affixed toa rigid mount plate. Cameras A through C and, optionally, other sensorsare affixed to a first rigid mount unit and cameras D through F and,optionally, other sensors are affixed to a second rigid mount unit,which are each affixed to a rigid mount plate. In particular, when theangular alignment of adjacent cameras affixed to the first or secondrigid mount units is improved, the angular alignment of the othersensors affixed to the mount unit is also enhanced. This enhancement ofalignment precision for the other sensors affixed to the rigid mountplate also improves the image resolution for those sensors.

Similarly, the horizontal hatched areas represent three image overlapareas 2112, 2116 and 2120. The forward, left image overlap area 2112 iswhere rear, right camera A overlaps with the forward, right camera D,forward, center image overlap area 2116 is where rear, center/nadircamera B overlaps with the forward, center camera E, and the rear, rightimage overlap area 2120 is where rear, left camera C overlaps withforward, left camera F.

Referring now to FIG. 21, the overlapping grid detail labeled “FORELAPAREA 4:1” represents overlapping forelap overlap areas 2112, 2116 and2120. In these forelap overlap areas 2112, 2116 and 2120, the camerasensor grid bisects each pixel in the overlap areas 2112, 2116 and 2120,which effectively quadruples the image resolution in these areas 2112,2116 and 2120 via the mechanism of co-mounted, co-registeredoversampling. In effect, the improvement in image/sensor resolution isdoubled in each dimension, or 2×2=4 times. This quadrupling of the imageresolution quadruples the alignment precision between correspondingcameras.

This quadrupling of alignment precision between corresponding camerasimproves the systems 100 alignment precision for all sensors affixed toa rigid mount plate. Cameras A through C and, optionally, other sensorsare affixed to a first rigid mount unit and cameras D through F and,optionally, other sensors are affixed to a second rigid mount unit,which are each affixed to a rigid mount plate. In particular, when theangular alignment of corresponding cameras affixed to the first orsecond rigid mount units is improved, the angular alignment of the othersensors is also enhanced. This enhancement of alignment precision forthe other sensors affixed to the rigid mount plate also improves theimage resolution for those sensors.

Similar to the overlapping sidelap overlap areas 2006, 2008 and 2014,2016 in FIG. 20, the intersecting forelap and sidelap overlap areas 2114and 2118 in FIG. 21 results in an astounding overall 64 timesimprovement in system calibration and camera alignment. Referring now toFIG. 21, the intersecting grid detail labeled “QUAD OVERLAP AREA 64:1”represents intersecting forelap and sidelap overlap area 2118. In theintersecting forelap and sidelap overlap areas 2114 and 2118, theoverlapping camera sensor grids bisects each pixel in the intersectingareas 2114 and 2118, which effectively quadruples the image resolutionin these areas 2114 and 2118 via the mechanism of co-mounted,co-registered oversampling. In effect, the improvement in image/sensorresolution is again doubled in each dimension, or 2×2×2×2×2×2=64 times.This overall 64 times improvement of the image resolution also enhancesalignment precision by 64 times between adjacent cameras.

This 64 times improvement of alignment precision between adjacent andcorresponding cameras enhances the systems 100 alignment precision forall sensors affixed to a rigid mount plate. Cameras A through C and,optionally, other sensors are affixed to a first rigid mount unit andcameras D through E and, optionally, other sensors are affixed to asecond rigid mount unit, which are each affixed to a rigid mount plate.In particular, when the angular alignment of adjacent and/orcorresponding cameras affixed to the first and/or second rigid mountunits is improved, the angular alignment of the other sensors is alsoenhanced. This enhancement of alignment precision for the other sensorsaffixed to the rigid mount plate also improves the image resolution forthose sensors.

In a preferred embodiment, one camera array is monochrome, and anothercamera array is red-green-blue. Even though each array covers differentcolor bands, simple image processing techniques are used so that allcolor bands realize the benefit of this increased resolution. Anotheradvantage provided by these techniques is that, in the case where onecamera array is red-green-blue and the other, overlapping camera arrayis an infrared or near infrared (or some other bandwidth), which resultsin a superior multi-spectral image.

As shown in FIGS. 19-21, these techniques may be used to overcome theresolution limits imposed on camera systems due to the inability ofoptical glass to resolve “very small” objects. In particular, there areknown physical limits to the ability of optical glass in camera lensesto resolve very small objects. This is often called “the resolving limitof glass”. For example, if 1 millimeter pixels are required from 10,000feet of altitude, the use of an extremely high magnification telescopiclens would be required to obtain a ground swath of about 100 feet. Thisis because no matter how many pixels can be produced by acharged-coupled device sensor (e.g., 1 billion pixels), the resolvingpower of the purest glass would not permit image resolution to 1millimeter pixels at 10,000 feet of altitude. This example is used tomake the point that there are physical limits for pixel resolution inglass as well as pixel density limits for an imaging sensor.

The systems 100 imaging sensor alignment in the rigid mount unit(s)affixed to the rigid mount plate and related calibration techniquesprovide a unique solution to this problem, as described above. By usingthese techniques, the resolving limitations of glass can effectively beovercome. For example, a single camera array results in 1 times (or no)oversampling benefits. However, two overlapping camera arrays results in4 times overall improvement in both image resolution and overallgeospatial horizontal and vertical accuracy. Further, three overlappingcamera arrays results in 16 times overall improvement, four overlappingcamera arrays results in 64 times overall improvement, and so on.

As can be deduced from these examples, the equation for overallimprovement is as follows:overall improvement=4^(N)where N is the number of overlapping camera arrays.

If there are four camera arrays, then there are three overlapping cameraarrays (i.e., N=3). Accordingly, four camera arrays provide a 64 times(i.e., 4³=64 times) overall improvements in both the image resolutionand overall geospatial horizontal and vertical accuracy.

Further, these subpixel calibration techniques may be combined with theself-locking flight path techniques, as disclosed in U.S. PublicationNo. 2004/0054488A1, now U.S. Pat. No. 7,212,938B2, the disclosure ofwhich is hereby incorporated by reference in full.

In addition to fore and/or lateral co-mounted, co-registeredoversampling as shown in FIGS. 19-21, the present invention may alsoemploy flight line oversampling as well to further improve the imageresolution, as shown in FIGS. 13-17. As shown in FIGS. 13-17, the flightlines overlap each other in an image region because each flight line isparallel to one another. These overlapping image regions may be used tocalibrate the sensors by along-track and cross-track parallax of imagesin adjacent flight lines using stereographic techniques.

In an embodiment, the self-locking flight path may comprise any patternthat produces at least three substantially parallel travel lines out ofa group of three or more travel lines. Further, at least one of thetravel lines should be in an opposing direction to the othersubstantially parallel travel lines. In a preferred embodiment, thetravel pattern comprises at least one pair of travel lines in a matchingdirection and at least one pair of travel lines in an opposingdirection.

When using the self-locking flight path in opposite directions, theobservable positional error may be doubled in some image regions.According, the self-locking flight path technique includes an algorithmto significantly reduce these positional errors. This reduction inpositional errors is especially important in the outside, or far leftand far right “wing” image areas where the greatest positional errorsoccur.

In an embodiment, these positional improvements may be realized by usinga pattern matching technique to automatically match a pixel pattern areaobtained from a flight line (e.g., North/South) with the same pixelpattern area obtained from an adjacent flight line (e.g., North/South).In a preferred embodiment, the latitude/longitude coordinates from oneor more GPS location systems may be used to accelerate this patternmatching process.

Similarly, these subpixel calibration and self-locking flight pathtechniques may be combined with stereographic techniques becausestereographic techniques rely heavily on the positional accuracy of eachpixel relative to all other pixels. In particular, these techniquesimprove the stereographic image resolution and overall geospatialhorizontal and vertical accuracy, especially, in the far left and farright “wing” image areas, where the greatest positional errors occur.Further, stereographic techniques are used to match known elevation datawith the improved stereographic datasets. Accordingly, the combinedsubpixel calibration, self-locking flight path and stereographictechniques provide a greatly improved Digital Elevation Model, whichresults in superior image.

Further, these subpixel calibration and self-locking flight pathtechniques may be used to provide a dynamic, RealTime calibration of thesystem 100. In particular, these techniques provide the ability torapidly “roll on” one or more camera array assemblies 112 onto thesystem 100, to immediately begin collecting image data of a target areaand to quickly produce high-quality images because the individualsensors have been initially calibrated in the rigid mount unit(s)affixed to the rigid mount plate, as discussed above. In particular, thecamera sensors are co-registered to calibrate the physical mount angleoffset of each sensor relative to each other and/or to the nadir camera.In an embodiment, multiple, i.e., at least two, rigid mount units areaffixed to a rigid mount plate and are co-registered. This provides aninitial, “close” calibration. These initial calibration parameters maybe entered into an onboard computer system 104 in the system 100, andupdated during flight using oversampling techniques, as discussed above.

In an embodiment, the system 100 comprises a RealTime, self-calibratingsystem to update the calibration parameters. In particular, the onboardcomputer 104 software comprises a RealTime software “daemon” (i.e., abackground closed-loop monitoring software) to constantly monitor andupdate the calibration parameters using the co-mounted, co-registeredoversampling and flight line oversampling techniques, as discussedabove. In a preferred embodiment, the RealTime daemon combines subpixelcalibration, self-locking flight path and stereographic techniques toimprove the stereographic image resolution and overall geo spatialhorizontal and vertical accuracy. In particular, stereographictechniques are used to match known elevation data to the improvedstereographic datasets. Accordingly, the combined subpixel calibration,self-locking flight path and stereographic techniques provide a greatlyimproved Digital Elevation Model, which results in superior image.

In an embodiment, the system 100 comprises a RealTime GPS data system toprovide GPS input data. Calibration accuracy is driven by input datafrom electronic devices such as a GPS and an IMU, and by calibrationsoftware which is augmented by industry standard GPS and IMU softwaresystems. Accordingly, a key component of this RealTime, self-calibratingsystem is a RealTime GPS input data via a potentially low bandwidthcommunication channel such as satellite phone, cell phone, RF modem, orsimilar device. Potential sources for the RealTime GPS input datainclude project controlled ad-hoc stations, fixed broadcast GPSlocations (or similar) or inertial navigation via an onboard IMU.

The modules, algorithms and processes described above can be implementedin a number of technologies and configurations. Embodiments of thepresent invention may comprise functional instances of software orhardware, or combinations thereof. Furthermore, the modules andprocesses of the present invention may be combined together in a singlefunctional instance (e.g., one software program), or may compriseoperatively associated separate functional devices (e.g., multiplenetworked processor/memory blocks). All such implementations arecomprehended by the present invention.

The embodiments and examples set forth herein are presented to bestexplain the present invention and its practical application and tothereby enable those skilled in the art to make and utilize theinvention. However, those skilled in the art will recognize that theforegoing description and examples have been presented for the purposeof illustration and example only. The description as set forth is notintended to be exhaustive or to limit the invention to the precise formdisclosed. Many modifications and variations are possible in light ofthe above teaching without departing from the spirit and scope of thefollowing claims.

What is claimed is:
 1. A system for generating a map of a target area,comprising: a global positioning receiver; an elevation measurementunit, adaptably mountable to a vehicle; a global positioning antenna,adaptably mountable to the vehicle; an attitude measurement unit,adaptably mountable to the vehicle; an imaging sensor system, adaptablymountable to the vehicle having a view of the target area, comprising: arigid mount unit having at least two imaging sensors disposed within themount unit, wherein a first imaging sensor and a second imaging sensoreach has a focal axis passing through an aperture in the mount unit,wherein the first imaging sensor generates a first image area comprisinga first data array of pixels and the second imaging sensor generates asecond image area comprising a second data array of pixels, wherein thefirst and second imaging sensors are offset to have a first imageoverlap area in the target area, wherein the first sensors image databisects the second sensors image data in the first image overlap area;and a computer in communication with the elevation measurement unit, theglobal positioning antenna, the attitude measurement unit, the firstimaging sensor, and the second imaging sensor; correlating at least aportion of the image areas from the first imaging sensor and the secondimaging sensor to a portion of the target area based on input from oneor more of: the elevation measurement unit, the global positioningantenna and the attitude measurement unit.
 2. The system of claim 1further comprising: a third imaging sensor disposed within the mountunit, wherein the third imaging sensor has a focal axis passing throughthe aperture in the mount unit, wherein the third imaging sensorgenerates a third image area comprising a third data array of pixels. 3.The system of claim 2, further comprising: a fourth imaging sensordisposed within the mount unit, wherein the fourth imaging sensor has afocal axis passing through the aperture in the mount unit, wherein thefourth imaging sensor generates a fourth image area comprising a fourthdata array of pixels, wherein the third and fourth imaging sensors areoffset to have a second image overlap area in the target area, whereinthe third sensors image data bisects the fourth sensors image data inthe second image overlap area.
 4. The system of claim 3, wherein a firstsensor array comprising the first and second image sensors and a secondsensor array comprising the third and fourth image sensors are offset tohave a third image overlap area in the target area, wherein the firstsensor arrays image data bisects the second sensor arrays image data inthe third overlap area.
 5. The system of claim 3, wherein the firstsensors arrays image data completely overlaps the second sensors arraysimage data.
 6. The system of claim 3, wherein third and fourth imagingsensors are selected from the group consisting of digital cameras,LIDAR, infrared, heat-sensing and gravitometers.
 7. The system of claim3, wherein the first and second imaging sensors are a digital camera andthe third imaging sensor is a LIDAR.
 8. The system of claim 2, whereinthe third imaging sensor is selected from the group consisting ofdigital cameras, LIDAR, infrared, heat-sensing and gravitometers.
 9. Thesystem of claim 2, wherein the third imaging sensor is selected from thegroup consisting of a digital camera having a hyperspectral filter and aLIDAR.
 10. The system of claim 2, wherein the first and second imagingsensors are a digital camera and the third imaging sensor is a LIDAR.11. The system of claim 1, wherein the mount unit flexes less than 100thof a degree during operation.
 12. The system of claim 11, wherein themount unit flexes less than 1,000th of a degree during operation. 13.The system of claim 12, wherein the mount unit flexes less than 10,000thof a degree during operation.
 14. The system of claim 1, wherein thefirst imaging sensor is calibrated relative to one or more attitudemeasuring devices selected from the group consisting of a gyroscope, anIMU, and a GPS.
 15. The system of claim 1, wherein the first and secondimaging sensors are selected from the group consisting of digitalcameras, LIDAR, infrared, heat-sensing and gravitometers.
 16. An imagingsensor system comprising: a mount unit adaptably mountable to a vehiclein alignment with a target area, having at least two imaging sensorsdisposed within the mount unit, wherein a first imaging sensor and asecond imaging sensor each has a focal axis passing through an aperturein the mount unit, wherein the first imaging sensor generates a firstimage area comprising a first data array of pixels and the secondimaging array generates a second image area comprising a second dataarray of pixels, wherein the first and second imaging sensors are offsetto have a first image overlap area in the target area, wherein the firstsensors image data bisects the second sensors image data in the firstimage overlap area.
 17. The system of claim 16 further comprising: athird imaging sensor disposed within the mount unit, wherein the thirdimaging sensor has a focal axis passing through the aperture in themount unit, wherein the third imaging sensor generates a third imagearea comprising a third data array of pixels.
 18. The system of claim 17further comprising: a fourth imaging sensor disposed within the mountunit, wherein the fourth imaging sensor has a focal axis passing throughthe aperture in the mount unit, wherein the fourth imaging sensorgenerates a fourth image area comprising a fourth data array of pixels,wherein the third and fourth imaging sensors are offset to have a secondimage overlap area in the target area, wherein the third sensors imagedata bisects the fourth sensors image in the second image overlap area.19. The system of claim 18, wherein a first sensors array comprising thefirst and the second image sensor and a second sensors array comprisingthe third and the fourth image sensor are offset to have a third imageoverlap area in the target area, wherein first sensor arrays image databisects the second sensor arrays image data in the third image overlaparea.
 20. The system of claim 18, wherein the first sensors arrays imagedata completely overlaps the second sensors arrays image data.
 21. Thesystem of claim 18, wherein the third and fourth imaging sensors areselected from the group consisting of digital cameras, LIDAR, infrared,heat-sensing and gravitometers.
 22. The system of claim 18, wherein thefirst and second imaging sensors are a digital camera and the thirdimaging sensor is a LIDAR.
 23. The system of claim 17, wherein the thirdimaging sensor is selected from the group consisting of digital cameras,LIDAR, infrared, heat-sensing and gravitometers.
 24. The system of claim17, wherein the third imaging sensor is selected from the groupconsisting of a digital camera having a hyperspectral filter and aLIDAR.
 25. The system of claim 17, wherein the first and second imagingsensors are a digital camera and the third imaging sensor is a LIDAR.26. The system of claim 16, wherein the mount unit flexes less than100th of a degree during operation.
 27. The system of claim 26, whereinthe mount unit flexes less than 1,000th of a degree during operation.28. The system of claim 27, wherein the mount unit flexes less than10,000th of a degree during operation.
 29. The system of claim 16,wherein the first imaging sensor is calibrated relative to one or moreattitude measuring devices selected from the group consisting of agyroscope, an IMU, and a GPS.
 30. The system of claim 16, wherein thefirst and second imaging sensors are selected from the group consistingof digital cameras, LIDAR, infrared, heat-sensing and gravitometers. 31.A method of calibrating imaging sensors comprising the steps of:performing an initial calibration of the imaging sensors comprising:determining the position of an AMU; determining the position of a firstimaging sensor within a rigid mount unit relative to the AMU;determining the position of a second imaging sensor within the rigidmount unit relative to the AMU; calibrating the first imaging sensoragainst a target area and determining a boresight angle of the firstimaging sensor; and calculating the position of one or more subsequentimaging sensors within the rigid mount unit relative to the firstimaging sensor; and calibrating the one or more subsequent imagingsensors using the boresight angle of the first imaging sensor; and usingoversampling techniques to update at least one initial calibrationparameter of the first imaging sensor against a target area and theboresight angle of the first imaging sensor; using oversamplingtechniques to update the position of one or more subsequent imagingsensors within the rigid mount unit relative to the first imagingsensor; and updating at least one calibration parameter of one or moresubsequent imaging sensors within the rigid mount using the updatedboresight angle of the first imaging sensor.
 32. The method of claim 31,wherein the initial calibration step further comprises the step of:calibrating the second imaging sensor using the updated boresight angleof the first imaging sensor.
 33. The method of claim 32, furthercomprising the step of: using oversampling techniques to update theposition of the second imaging sensor within the rigid mount unitrelative to the first imaging sensor.
 34. The method of claim 31,further comprising the steps of: using flight line oversamplingtechniques to update the calibration of the first imaging sensor againsta target area and the boresight angle of the first imaging sensor; andusing flight line oversampling techniques to update the position of oneor more subsequent imaging sensors within the rigid mount unit relativeto the first imaging sensor.
 35. The method of claim 34, furthercomprising the step of: using flight line oversampling techniques toupdate the position of the second imaging sensor within the rigid mountunit relative to the first imaging sensor.
 36. A system for generating amap of a surface, comprising: a global position receiver; an elevationmeasurement unit, adaptably mountable to a vehicle; a global positioningantenna, adaptably mountable to the vehicle; an attitude measurementunit, adaptably mountable to the vehicle; an imaging array, having aview of the surface, comprising: a mount unit, adaptably mountable tothe vehicle; an aperture, formed in the mount unit; a first imagingsensor, coupled to the mount unit, having a first focal axis passingthrough the aperture, wherein the first image sensor generates a firstimage area of the surface comprising a first data array of pixels,wherein the first data array of pixels is at least two dimensional; anda second imaging sensor, coupled to the mount unit and offset from thefirst imaging sensor, having a second focal axis passing through theaperture and intersecting the first focal axis, wherein the secondimaging sensor generates a second image area of the surface comprising asecond data array of pixels, wherein the second data array of pixels isat least two dimensional; and a computer, connected to the elevationmeasurement unit, the global positioning antenna, the attitudemeasurement unit and first and second imaging sensors; correlating atleast a portion of the image area from the first and second imagingsensors to a portion of the surface based on input from one or more of:the elevation measurement unit, the global positioning antenna and theattitude measurement unit.
 37. The system of claim 36, furthercomprising a third imaging sensor, coupled to the mount unit and offsetfrom the first imaging sensor, having a third focal axis passing throughthe aperture and intersecting the first focal axis within anintersection area.
 38. The system of claim 37, wherein the focal axes ofthe third imaging sensor lies in a common plane with the focal axes ofthe first and second imaging sensors.
 39. The system of claim 37,wherein the focal axes of the first and second imaging sensors lie in afirst common plane and the focal axis of the third imaging sensor liesin a plane orthogonal to the first common plane.
 40. A system forgenerating a map of a surface, comprising: a global position receiver;an elevation measurement unit, adaptably mountable to a vehicle; aglobal positioning antenna, adaptably mountable to the vehicle; anattitude measurement unit, adaptably mountable to the vehicle; a firstimaging sensor, adaptably mountable to the vehicle, having a view of thesurface, having a focal axis disposed in the direction of the surface,wherein the first imaging sensor generates an image area comprising afirst data array of pixels, wherein the first data array of pixels is atleast two dimensional; and a computer, connected to the elevationmeasurement unit, the global positioning antenna, the attitudemeasurement unit and the first imaging sensor; generating a calculatedlongitude and calculated latitude value for a coordinate correspondingto at least one pixel in the array based on input from one or more of:the elevation measurement unit, the global positioning antenna and theattitude measurement unit.
 41. A system for generating a map of a targetarea, comprising: a global position receiver; an elevation measurementunit, adaptably mountable to the vehicle; a global positioning antenna,adaptably mountable to the vehicle; an attitude measurement unit,adaptably mountable to the vehicle; an imaging sensor system, having aview of the target area, comprising: a mount unit, adaptably mountableto the vehicle, having a first and second imaging sensor disposed withinthe mount unit, wherein the first and second imaging sensors each have afocal axis passing through an aperture in the mount unit, wherein thefirst imaging sensor generates a first image area comprising a firstdata array of pixels and second imaging sensor generates a second imagearea comprising a second data array of pixels, wherein the first andsecond data array of pixels is at least two dimensional; and a computerin communication with the elevation measurement unit, the globalpositioning antenna, the attitude measurement unit, the first imagingsensor, and the second imaging sensor; correlating at least a portion ofthe image area from the first imaging sensor and the second imagingsensor to a portion of the target area based on input from one or moreof: the elevation measurement unit, the global positioning antenna andthe attitude measurement unit.
 42. The system of claim 41, furthercomprising a third imaging sensor disposed within the mount unit,wherein the third imaging sensor has a focal axis passing through anaperture in the mount unit, wherein the third imaging sensor generates athird image area comprising a third data array of pixels.
 43. An imagingsensor system comprising: a mount unit, adaptably mountable to avehicle, having a first and second imaging sensors disposed within themount unit, wherein the first imaging and second imaging sensors eachhave a focal axis passing through an aperture in the mount unit, whereinthe first imaging sensor generates a first image area comprising a firstdata array of pixels and the second imaging sensor generates a secondimage area comprising a second data array of pixels, wherein the firstand second data array of pixels is at least two dimensional.
 44. Thesystem of claim 43, further comprising a third imaging sensor disposedwithin the mount unit, wherein the third imaging sensor has a focal axispassing through an aperture in the mount unit, wherein the third imagingsensor generates a third image area comprising a third data array ofpixels.